Automatic Partitioning of MPI Operations in MPI+OpenMP Applications

The new MPI 4.0 standard includes a new chapter about partitioned point-to-point communication operations. These partitioned operations allow multiple actors of one MPI process (e.g. multiple threads) to contribute data to one communication operation. These operations are designed to mitigate current problems in multithreaded MPI programs, with some work suggesting a substantial performance benefit (up to 26%) when using these operations compared to their existing non-blocking counterparts. In this work, we explore the possibility for the compiler to automatically partition sending operations across multiple OpenMP threads. For this purpose, we developed an LLVM compiler pass that partitions MPI sending operations across the different iterations of OpenMP for loops. We demonstrate the feasibility of this approach by applying it to 2D stencil codes, observing very little overhead while the correctness of the codes is sustained. Therefore, this approach facilitates the usage of these new additions to the MPI standard for existing codes. Our code is available on github: https://github.com/tudasc/CommPart

Compiler-enabled optimization of persistent MPI Operations

MPI is widely used for programming large HPC clusters. MPI also includes persistent operations, which specify recurring communication patterns. The idea is that the usage of those operations can result in a performance benefit compared to the standard non-blocking communication. But in current MPI implementations, this performance benefit is not really observable. We determine the message envelope matching as one of the causes of overhead. Unfortunately, this matching can only hardly be overlapped with computation. In this work, we explore how compiler knowledge can be used to extract more performance benefit from the usage of persistent operations. We find that the compiler can do some of the required matching work for persistent MPI operations. As persistent MPI requests can be used multiple times, the compiler can, in some cases, prove that message matching is only needed for the first occurrence and can be entirely skipped for subsequent instances. In this paper, we present the required compiler analysis, as well as an implementation of a communication scheme that skips the message envelope matching and directly transfers the data via RDMA instead. This allows us to substantially reduce the communication overhead that cannot be overlapped with computation. Using the Intel IMB-ASYNC Benchmark, we can see a communication overhead reduction of up to 95 percent for larger message sizes

MPI-CorrBench: Towards an MPI Correctness Benchmark Suite

The Message Passing Interface (MPI) is the de-facto standard for distributed memory computing in high-performance computing (HPC). To aid developers write correct MPI programs, different tools have been proposed, e.g., Intel Trace Analyzer and Collector (ITAC), MUST, Parcoach and MPI-Checker. Unfortunately, the effectiveness of these tools is hard to compare, as they have not been evaluated on a common set of applications. More importantly, well-known and widespread benchmarks, which tend to be well-tested and error free, were used for their evaluation. To enable a structured comparison and improve the coverage and reliability of available MPI correctness tools, we propose MPI-CorrBench as a common test harness. MPI-CorrBench enables a structured comparison of the different tools available w.r.t. various types of errors. In our evaluation, we use MPI-CorrBench to provide a well-defined set of error-cases to MUST, ITAC, Parcoach and MPI-Checker. In particular, we find that ITAC and MUST complement each other in many cases. In general, MUST works better for detecting type errors while ITAC is better in detecting errors in non-blocking operations. Although the most-used functions of MPI are well supported, MPI-CorrBench shows that for one sided communication, the error detection capability of all evaluated tools needs improvement. Moreover, our experiments reveal a MPI standard violation in the MPICH test suite as well as several cases of discouraged use of MPI functionality

Investigating the Usage of MPI at Argument-Granularity in HPC Codes

This study focuses on gaining insights into the usage of the Message-Passing Interface (MPI) in a large set of High-Performance Computing (HPC) codes by analyzing MPI function calls and their argument usage patterns. Previous work has focused on analyzing MPI feature usage by statically matching function calls. However, this approach does not reveal common argument-specific call patterns or cross-interactions between MPI functions. In particular, MPI exposes its internal data structures using handles, and users pass these handles to MPI constructor functions, e.g., to create custom communicators. Tracking the relevant MPI arguments of these constructors and cross-referencing them with other MPI calls in a target code can reveal common user interactions. These insights can be used to optimize, e.g., datatype construction at a library level or to extend MPI correctness debugging tools to verify correct construction of these data structures. To that end, we statically analyze codes to extract MPI function calls and their arguments, cross-reference them with other MPI calls, and provide statistics on common argument patterns and cross-use of MPI functions. We believe that these insights can guide further development within the MPI community to ultimately benefit users

Towards a Hybrid MPI Correctness Benchmark Suite

High-performance computing codes often combine the Message-Passing Interface (MPI) with a shared-memory programming model, e.g., OpenMP, for efficient computations. These so-called hybrid models may issue MPI calls concurrently from different threads at the highest level of MPI thread support. The correct use of either MPI or OpenMP can be complex and error-prone. The hybrid model increases this complexity even further. While correctness analysis tools exist for both programming paradigms, for hybrid models, a new set of potential errors exist, whose detection requires combining knowledge of MPI and OpenMP primitives. Unfortunately, correctness tools do not fully support the hybrid model yet, and their current capabilities are also hard to assess. In previous work, to enable structured comparisons of correctness tools and improve their coverage, we proposed the MPI-CorrBench test suite for MPI. Likewise, others proposed the DataRaceBench test suite for OpenMP. However, for the particular error classes of the hybrid model, no such test suite exists. Hence, we propose a hybrid MPI-OpenMP test suite to (1) facilitate the correctness tool development in this area and, subsequently, (2) further encourage the use of the hybrid model at the highest level of MPI thread support. To that end, we discuss issues with this hybrid model and the knowledge correctness tools need to combine w.r.t. MPI and OpenMP to detect these. In our evaluation of two state-of-the-art correctness tools, we see that for most cases of concurrent and conflicting MPI operations, these tools can cope with the added complexity of OpenMP. However, more intricate errors, where user code interferes with MPI, e.g., a data race on a buffer, still evade tool analysis

Postoperative continuous positive airway pressure to prevent pneumonia, re-intubation, and death after major abdominal surgery (PRISM): a multicentre, open-label, randomised, phase 3 trial

BackgroundRespiratory complications are an important cause of postoperative morbidity. We aimed to investigate whether continuous positive airway pressure (CPAP) administered immediately after major abdominal surgery could prevent postoperative morbidity.MethodsPRISM was an open-label, randomised, phase 3 trial done at 70 hospitals across six countries. Patients aged 50 years or older who were undergoing elective major open abdominal surgery were randomly assigned (1:1) to receive CPAP within 4 h of the end of surgery or usual postoperative care. Patients were randomly assigned using a computer-generated minimisation algorithm with inbuilt concealment. The primary outcome was a composite of pneumonia, endotracheal re-intubation, or death within 30 days after randomisation, assessed in the intention-to-treat population. Safety was assessed in all patients who received CPAP. The trial is registered with the ISRCTN registry, ISRCTN56012545.FindingsBetween Feb 8, 2016, and Nov 11, 2019, 4806 patients were randomly assigned (2405 to the CPAP group and 2401 to the usual care group), of whom 4793 were included in the primary analysis (2396 in the CPAP group and 2397 in the usual care group). 195 (8·1%) of 2396 patients in the CPAP group and 197 (8·2%) of 2397 patients in the usual care group met the composite primary outcome (adjusted odds ratio 1·01 [95% CI 0·81–1·24]; p=0·95). 200 (8·9%) of 2241 patients in the CPAP group had adverse events. The most common adverse events were claustrophobia (78 [3·5%] of 2241 patients), oronasal dryness (43 [1·9%]), excessive air leak (36 [1·6%]), vomiting (26 [1·2%]), and pain (24 [1·1%]). There were two serious adverse events: one patient had significant hearing loss and one patient had obstruction of their venous catheter caused by a CPAP hood, which resulted in transient haemodynamic instability.InterpretationIn this large clinical effectiveness trial, CPAP did not reduce the incidence of pneumonia, endotracheal re-intubation, or death after major abdominal surgery. Although CPAP has an important role in the treatment of respiratory failure after surgery, routine use of prophylactic post-operative CPAP is not recommended.<br/

Postoperative continuous positive airway pressure to prevent pneumonia, re-intubation, and death after major abdominal surgery (PRISM): a multicentre, open-label, randomised, phase 3 trial

Background: Respiratory complications are an important cause of postoperative morbidity. We aimed to investigate whether continuous positive airway pressure (CPAP) administered immediately after major abdominal surgery could prevent postoperative morbidity. Methods: PRISM was an open-label, randomised, phase 3 trial done at 70 hospitals across six countries. Patients aged 50 years or older who were undergoing elective major open abdominal surgery were randomly assigned (1:1) to receive CPAP within 4 h of the end of surgery or usual postoperative care. Patients were randomly assigned using a computer-generated minimisation algorithm with inbuilt concealment. The primary outcome was a composite of pneumonia, endotracheal re-intubation, or death within 30 days after randomisation, assessed in the intention-to-treat population. Safety was assessed in all patients who received CPAP. The trial is registered with the ISRCTN registry, ISRCTN56012545. Findings: Between Feb 8, 2016, and Nov 11, 2019, 4806 patients were randomly assigned (2405 to the CPAP group and 2401 to the usual care group), of whom 4793 were included in the primary analysis (2396 in the CPAP group and 2397 in the usual care group). 195 (8\ub71%) of 2396 patients in the CPAP group and 197 (8\ub72%) of 2397 patients in the usual care group met the composite primary outcome (adjusted odds ratio 1\ub701 [95% CI 0\ub781-1\ub724]; p=0\ub795). 200 (8\ub79%) of 2241 patients in the CPAP group had adverse events. The most common adverse events were claustrophobia (78 [3\ub75%] of 2241 patients), oronasal dryness (43 [1\ub79%]), excessive air leak (36 [1\ub76%]), vomiting (26 [1\ub72%]), and pain (24 [1\ub71%]). There were two serious adverse events: one patient had significant hearing loss and one patient had obstruction of their venous catheter caused by a CPAP hood, which resulted in transient haemodynamic instability. Interpretation: In this large clinical effectiveness trial, CPAP did not reduce the incidence of pneumonia, endotracheal re-intubation, or death after major abdominal surgery. Although CPAP has an important role in the treatment of respiratory failure after surgery, routine use of prophylactic post-operative CPAP is not recommended